Statistical Evaluation Method for Comparing Grid Map Based Sensor Fusion Algorithms

In this paper we present a method for evaluating sensor fusion algorithms based on a quantitative comparison, which is independent of the data acquired and the sensors used. The sensor fusion performance measures and performance analysis procedure provide a basis for modeling, analyzing, experimenting, and comparing different sensor fusion algorithms. The capability to compare different algorithms creates a ranking basis, making it possible to select the best algorithm. The statistical evaluation method defines the experimental design and statistical analysis. The numbers of experiments and repetitions required are derived from the statistical characteristics and the desired confidence level. Since procedures are defined to ensure that the experiments are indeed conducted differently, the results are not specific for either the evaluated test cases or the sensor characteristics. The statistical analysis provides a systematic method for comparing sensor fusion algorithms. Although this method requires experimentation, it offers the ability to compare actual performances in the real world. Quantitative procedures are developed to ensure that specific environmental conditions evaluated do not influence the evaluation. To demonstrate the statistical evaluation method it is applied to a case study that compared five different sensor fusion algorithms in a mobile robot experiment. KEY WORDS—evaluation method, mobile robots, performance measures, sensor fusion